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(57) Abstract: Primary computers (2-i) each have a ver- 
sion of a software to be tested, a local agent (23 1), capable 
of having commands to be run in the local agent's primary 
computer, and interfaces (25A-25C), for communication 
with a master computer (1). A representation of test op- 
erations is stored in the master computer (1). Commands 
derived from the test operations are dynamically and se- 
lectively sent from the master computer (1) to the local 
agent(s) (231), as designated by a name and a port num- 
ber. Java virtual machines (230) may be used in the pri- 
mary computers. 
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Method of and system for testing software in computers 

5 

This invention relates to computer technology. More particu- 
larly , it deals with testing the compatibility of software 
with various computer architectures . 

10 A new release of a software may comprise a particular version 
for each computer architecture ( "target " ) to be supported. 
Each such version has to be tested for full operability 
within the corresponding target architecture. Generally, this 
implies a number of operations: firstly, a separate software 

15 test program must be written and stored for each type of 
architecture, and of course for each software version; 
secondly, these test programs have to be copied by a test 
operator from a movable medium storage, or across a network 
connection, into the respective targets provided- with the 

20 software to be tested; thirdly, the test programs are 
executed one by one; finally, the result reports or logs have 
to be recovered from the targets back to the testing facility 
for common archival and further analysis* This method of 
testing is thus fairly complex and time-consuming. 

25 

In fact, the complexity of this testing process is increased 
because: a) in general, several successive provisional 
releases are prepared before the final software release, and 
b) very often, the software is prepared for several operating 
30 systems (including different versions of the same OS), for at 
least some of the hardware/software combinations . 

The testing process complexity is still further increased 
when the software to be tested is installed in a computer 
35 network comprising computer targets with different 
architectures, and has networking functions, since the 
operation between each possible combination of targets also 
has to be tested. 

40 So, it is desirable to reduce the time and the number of 
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- an instruction for storing data in association with a local 
agent ; 

- an instruction for sending at least a portion of the 
software to be tested to a local agent, for storage in the 

5 primary computer. 

In an interesting embodiment, which is not limitative, the 
master computer and the target (s) are equipped with a common 
environment, preferably defining at least one Java virtual 
10 machine in each computer, and the local agent (s) is (are) 
written for this common environment. 

The indention also extends to the software code of a test 
program in a master test computer, and to the software code 
15 of a local agent. 

Other alternative features and advantages of the invention 
will appear in the detailed description below and in the 
appended drawings, in which : 

20 

- figure 1 is a general diagram of a computer network in 
which the invention is applicable; 

- figure 2 diagrammatically shows the organization of a 
25 primary computer or target 2 of figure 1; 

- figure 3 diagrammatically shows the memory organization of 
the master computer 1 of figure 1; 

30 - figure 4 is a flow-chart of a local agent in a target 2; 

- figure 5 shows the test supporting files required for 
creating a specific test environment in a master computer; 

35 - figure 6 shows how the successive instructions of a script 
in the master computer are executed; 

- figure 7 is a flow-chart of the execution of a single 
instruction in the master computer; and 
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- figure 8 is a flow-chart including the interaction of the 
master computer with the test personnel. 

Making reference to software entities imposes certain conven- 
tions in notation. In the detailed description: 

- the quote sign " may be used as character string delimiter 
when deemed necessary for clarity (e.g. " /usr/bin/ java" ) , 

- square brackets may be used to frame the optional portion 
of an entity or name (e.g. "[ , VALUE] " ) . 

In the following specif ication / reference will be made to the 
terms "station" or "host". These terms must be considered as 
involving a computer. In turn, "computer" means a physical 
machine, or a virtual machine. 

Considering a computer (physical or virtual machine) , the 
expression "software and hardware architecture" refers mainly 
to the operating system (OS) and to the motherboard , 
respectively. 

The invention may apply to at least one primary computer or 
target 2-i (i = 1 to N) , and to at least one master computer 
1 or "harness". The example in the foregoing description 
considers only one harness computer; however, several 
harnesses might be used simultaneously as well. Similarly, 
several targets are considered in the example, although only 
one target might be used as well. Furthermore, at least in 
certain situations, the harness (es) and target (s) may be 
hosted in the same physical machine. 

The harness 1 and targets 2^1 are preferably interconnected 
through a network, as illustrated in Fig. 1, where i 5=5 3. 
However, this networking feature is not absolutely necessary, 
since e.g. the harness and the agents may be run on the same 
physical machine. Additionally, both the harness (es) and the 
target (s) may be locally resident or they may be distributed 
(e.g. as with the server and client relationship) where 
communication between the two takes place over some sort of 
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network connection e.g. LAN, Internet, etc. 

The expression "release version" hereinafter means a new 
release of a software, in a version adapted to a particular 
5 computer architecture (and programming language, if appro- 
priate), or more generally any version modified for some 
reason . 

As shown in Fig. 1, harness 1 has a central processing unit 
10 or CPU 11, a RAM 13 (or any other working memory, for example 
DRAM, SDRAM or flash memory), and a hard disk 14 (or any 
other mass memory, for example floppy disks, removable 
drives, or flash memory emulating a disk). It may also have 
input/output devices like a display 15, a keyboard 19, and a 
15 pointer e.g. a mouse 18. A graphic user interface (GUI) may 
also be included if desired. The interconnections within 
harness 1 are diagrammatically shown as a single bus 10, 
although a more complex bus structure is generally used. 

20 The architecture of harness 1 is mainly defined by its CPU 11 
and its operating system (OS), stored e.g. in hard disk 14. 
The OS may be, for example, Solaris (a registered trademark 
of SUN MICROSYSTEMS) or Windows NT (a registered trademark of 
MICROSOFT) . 

25 

Harness 1 further has TCP sockets 17-1 through 17-3, for 
selective communication with corresponding TCP sockets 25-1 
through 25-3 of respective targets 2-1 through 2-3 (more 
generally targets 2-i, with i = 3 on Fig. 1). Other 
30 communication interfaces may be used in lieu of TCP sockets. 

Each target 2-i (Fig. 2) comprises a CPU 21, a RAM 23 (or any 
other working memory, for example DRAM, SDRAM or flash 
memory), a hard disk 24 (or any other mass memory, for 
35 example floppy disks, removable drives, or flash memory 
emulating a disk) and optional peripherals (not shown), which 
may include dedicated equipment, such as network adapters, 
removable storage media, or pluggable hardware components. A 
target OS is stored e.g. in hard disk 24. A local file system 
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(LFS) may be organized in hard disk 24 ("local" here means 
visible to the target, but possibly residing elsewhere, as 
e.g. in a Network File System or NFS scenario). This is 
exemplary only, and the invention may apply to other hardware 
5 target structures. 

As shown in Fig. 1, the targets may also comprise communica- 
tion equipment enabling them to be interconnected in a 
network via a medium 9, which may be comprised by e.g. 
0 Ethernet or token-ring running e.g. on fiber optics or 
twisted pair cabling or radio transmission or infra-red 
transmission. Harness 1 may also be connected to the network. 

The targets may have various architectures in terms of 
hardware and/or software. For example, the target architec- 
ture may be a SOLARIS -INTEL, a SOLARIS-SPARC, a WINDOWS NT- 
INTEL, a WINDOWS NT-ALPHA or a WINDOWS 95-INTEL one. 
(SOLARIS, SPARC are registered trademarks of SUN 
MICROSYSTEMS; INTEL is a registered trademark of INTEL Corp.; 
0 WINDOWS 95 and WINDOWS NT are registered trademarks of 
MICROSOFT; ALPHA is a registered trademark of DIGITAL 
EQUIPMENT). The targets may range e.g. from notebooks to 
desktop stations • 

In the example shown in Fig. 2, the target 2 has in RAM a 
Java virtual machine or JVM 230, which in turn hosts a local 
agent 231. The local agent 231 has a TCP server socket 25 to 
accept connection from the harness, as necessary. These 
connections are created as requested by the harness, e.g. as 
illustrated in 25A-17A to 25C-17C. 

It is preferable, although not necessary, that the operating 
systems of both the harness 1 and the targets 2-i be equipped 
with a common environment, which the local agents are written 
for. In this particular example , the Java environment is 
discussed. However, the invention is not so limited. Thus, as 
shown in Fig. 3, the hard disk 14 of harness 1 contains OS 
files 140, Java supporting files 141, and files 8 supporting 
the software tests. The Java files 141 are used to implement 
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a Java virtual machine 130 which is executed in the RAM 13 of 
machine 1, and hosts a "logical" harness 131, which is in 
fact a program running the test for the target(s) remotely, 
from the harness, using test supporting files 8, and sockets 
17-i. 

In each target, the local agent 231 acts as a kind of "mini- 
server", capable of communicating with harness 1 on request 
from this harness, and of running commands in its hosting 
target 2. Concretely, the local agent is a program written in 
a language adapted to the target architecture, and stored in 
its memory 23. The language may be Java when the target 
implements a Java virtual Machine. Alternatively, other 
languages may be used, according to the target architecture 
and environment . 

This local agent program 231 (more briefly "local agent") is 
arranged for managing the local target TCP socket 25. This 
enables communication between the target local agent 231 and 
the harness 1, independently of other target local agents. 
So, each local agent is attached to one TCP socket 25 of its 
hosting target computer 2. As illustrated in Fig. 2, if three 
local agents are implemented in the same target computer, 
that target computer will have three TCP sockets 25A through 
25C, each of which may be implemented through a server 
socket. Although each local agent is preferably hosted in its 
own JVM, this is not shown in the simplified representation 
of Fig. 2. 

The basic operation of a local agent is shown in the flow 
chart of Fig. 4. Normally (step 40), a local agent waits for 
receipt of a request from the harness, on its dedicated TCP 
socket. Upon such a receipt, step 42 may optionally check 
e.g. the syntax and/or arguments of the request and its 
arguments, at least partially. If the request is found not 
consistent (at 44), then an error is returned to the harness 
at step 45, and control is given to step 40. Otherwise 
(normally), step 4 7 launches execution of the request in the 
local target ; a report, together with results if 
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appropriate, is returned to the harness at step 49, and 
control is returned to step 40 of waiting for another request 
from the harness. 

5 The program for such a harness and local agent interaction 
may be written e.g. in Java, or in any other desired 
language, including Interface Definition Language ( IDL ) of 
the Object Management Group (OMG), and converted into an 
executable version in the target. A number of functionalities 
10 may have to be implemented in the local agent, depending upon 
the set of harness commands to be executed in the target, as 
described hereinafter. Programming each of these 
functionalities is considered accessible to men skilled in 
the art. 

15 

For purposes of software testing, each target 2-i is provided 
with a copy, or a portion of a copy, of the software to be 
tested, in a release version matching its own architecture 
(as above defined), and also matching its supported 
20 programming languages, if appropriate. 

The release version may be previously installed in the mass 
memory 24 of the target, or selectively sent from the harness 
1 to the target 2-i e.g. through the network, just before 
25 running a test. The release version may be an application, 
which may involve information obtained through communication 
with other computers, or from local files stored in the mass 
memory or hard disk 24 . 

30 Harness 1 is used for selectively distributing specific test 
sequences to the targets 2-i, in which respective release 
versions have to be tested. As mentioned before, the specific 
sequences are basically intended for selecting the release 
versions to correctly operate within the target architec- 

35 tures. 

For this purpose, harness 1 has test supporting files 8 
stored in its hard disk 14. In the example of Fig. 5, the 
test supporting files 8 firstly comprise agent definitions 
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81, which comprise agent variables (or agent properties , for 
e.g. as given in Example A.l in Exhibit A). For a given local 
agent hosted in a given target , the agent properties include 
at least a portion of the information needed at the harness 
level for sending that local agent a request being executable 
in the target. 

The test supporting files 8 further comprise a generic test 
section 84/ having, in the example: 

- target OS specific test code, e.g. Solaris specific test 
code 841S and NT specific test code 841N; 

- optionally, versions of the software to be tested for 
various target architectures, e.g. a Solaris version 842S and 
an NT version 842N; and 

- generic test code 850, examples of which will be given 
hereinafter. 

The test supporting files 8 may further comprise optional 
default settings 82, corresponding e.g. to standard system 
directories as used with Solaris or NT. 

Seen from harness 1, each local agent is preferably identi- 
fied by a unique couple or doublet of parameters, comprising 
a name which identifies the local agent (and, hence, its host 
target), and a port number which identifies its TCP socket. 
While this is an Internet Protocol (IP) type identification, 
other non IP identifications might be used as well. This 
couple may be the only information known by the harness about 
the local agent, i.e. all the other information about the 
local agent will be determined at the time of executing the 
test, by querying each target for more detailed information 
through appropriate usage of the test script code, as 
illustrated in an example on Figure 6. 

Generally, a local agent 231 will execute commands sent from 
harness 1. Such a command may result in an action in the 
target computer hosting the local agent; the action may be 
internal, or comprise sending file data required by harness 
1, or sending file data required (on command) by another 
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local agent, to be transmitted to harness 1; data represen- 
ting the test result in the target computer 2 hosting the 
local agent may also be returned to harness 1 . 

5 In accordance with this invention , a representation of a 
sequence of test operations is stored in harness 1. Then 
harness 1 will selectively send commands derived from at 
least certain of the test operations , to a local agent- The 
selection of commands may be based on available information 
10 about the target agent. At least part of the available 
information may be dynamically acquired. 

When sent to a given local agent, the sequence of test 
operations has to be configured in accordance with the 
15 specific environment of the target 2-i hosting that given 
local agent, including its operating system (OS) and hardware 
architecture . 

In accordance with another aspect of this invention (Fig. 6), 
20 the sequences of test operations are loaded in RAM 13, e.g. 
from one or more files. This forms a script 60, which is read 
by a harness test parser 64 also loaded in RAM 13, taking 
into account the current designated local agent (agent name 
and agent port number i.e. the agent doublet of parameters). 
25 Fig. 6 also shows auxiliary data files 62, for use in 
implementing the script, and stored in hard disk 14. The 
parser 64 accesses the instructions in the script 60 one by 
one. The result of the parsing, at 66, is a sequence of one 
or more commands, which are now in an executable form. The 
30 number of commands in the sequence depends upon the design of 
the basic script file 60, and upon the interaction Of the 
user with parser 64. The commands are then executed at 70. 

With reference to Fig. 7, the execution 70 starts with a 
35 first command as the current command, which is executed at 
71. If appropriate (depending upon whether the command is at 
least partially executable on the target), the command is 
sent as a request to the local agent being involved. If the 
command is not successfully executed (72), an error 
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processing is made at 73. Otherwise (normally) , if step 74 
indicates further commands, step 75 goes to the next command, 
and control is given to step 71. Step 79 is the end of the 
processing of the sequence of commands. 

5 

Thus, a test sequence module, comprising a representation of 
a sequence of test operations, is formed on the basis of the 
test supporting files 8, used as desired by the script 60. 
The parser 64, its products 66 (a working area in RAM), and 
10 the execution module 70 together act as a test control 
module, based on the script 60. 

Preferably, instead of being written in a form being directly 
executable in the target, the instructions in the test 

15 sequences refer to parameters (or variables), which are 
defined later. Some of these variables may be file names, 
thus enabling to select a file depending upon the target 
architecture. This is made preferably by using a "master 
language" (a language adapted to control the test from the 

20 harness) to write these instructions ( "master instructions" ) . 
The instructions in the master language are e.g. interpreted 
at the time of their execution, i.e. at step 70 latest. 

Non limitative examples of master language instructions will 
25 now be given. 

GETFILE(AGENT_ID,SRC_PATH,DESTJPATH) is a request instruction 
for retrieving an arbitrary file from a local agent 
designated by AGENT_ID, in its path SRCJPATH, and sending it 
30 to path DEST_PATH, relative to the harness 1. 

SENDFILE(AGENT_ID, SRCJPATH, DEST_PATH) is a request instruc- 
tion for sending an arbitrary file to a local agent AGENT_ID, 
from a path SRC_PATH, relative to the harness 1 , to a path 
35 DEST_PATH relative to the target having the local agent 
designated by AGENT_ID. 

In case of local agents running on various targets 2-i, it is 
advantageous to use target independent methods for specifying 



N.<annrtm- *wn 



ni71S02A1 I > 



WO 01/71502 PCT/IB00/00338 

13 

the paths. The following examples are merely exemplary. 

SETAGENT ( AGENT_ID , AGENT_NAME / AGENTJPORT , WORK_DIR ) is a master 
instruction which sets up a local agent for use during the 
5 lifetime of a current master instruction file. In the argu- 
ments: 

* AGENT_ID (return parameter) designates the local agent; 

* AGENT_ NAME designates the Java virtual machine running the 
designated local agent; 

10 * AGENT_PORT designates the port number of the designated 
local agent; 

* WORK_DIR designates a directory to place any files sent to 
the designated local agent via a ZIPSEND command (defined 
hereafter) • 

15 

ZIPSEND(AGENT_ID,ZIP_ID,ZIP_FILENAME) is a master instruc- 
tion/ which may be defined with three arguments: 

* AGENT_ID designates the local agent to send the zip file to 
(the local agent has been created by a previous call to 

20 SETAGENT) ; 

* ZIP_ ID (return argument only) designates the handle to the 
zip file who will be created. So, it must be undefined at 
this point and will be used thereafter when a call is made to 
a ZIPDEL command (delete a previously sent ZIPFILE); 

25 * Z I P_F I LENAME designates the full path to the zip file to be 
sent, relative to the harness 1. 

ZIPSEND will send the zip file ZiPJFILENAME from the harness 
to a local agent designated in AGENT_ID and request that 
30 agent to unzip the file. The unzipped file is placed in the 
directory designated by WQRK_DIR in the SETAGENT instruction 
for that AGENTJCD. 

SETPROP (AGENT_ID, KEY [, VALUE ] ) is a master instruction which 
35 sets a property value on a specified AGENT_ID, which will be 
visible to any subsequent CMD START commands (defined hereaf- 
ter) called on this AGENT_ID. In the arguments: 
* AGENT_ID which designates the local agent on which the 
property must be set; 
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* KEY which designates the key name of the property; 

* VALUE (optional third argument) designates the value of the 
property. If this argument is omitted, it will be retrieved 
from the specific chosen syntax environment, and if it is not 

5 defined here either, it will be assigned to the same value as 
KEY. 

For example, SETPROP(myAgeht, "System Root M , "c: \winnt" ) sets 
the "System Root" property of local agent "myAgent" to the 
10 value "c:\winnt". 

INCLUDE (OTHER_FILE) is a master instruction used to include 
another master instruction file OTHER_FILE into the current 
file. 

15 

CMDSTART(AGENT_ID,CMD_ID,CMD_STRING) is a request instruction 
which spawns an execute process on a local agent, using an 
environment built from any previous calls to SETPROP, and 
then executing the command specified in CMD_STRING (defined 
20 hereafter). It accepts at least three arguments: 

* AGENT_ID which designates the local agent on which the 
command is to be executed; 

* CMD_ID which designates a handle to be used for further 
communication of the started process; 

25 * CMD_STRING which designates the command to be executed on 
the designated local agent. It must match the OS and hardware 
architecture of the target hosting the local agent. 

REPORTSTATUS ( STATUS_CODE ) is a request instruction used to 
30 ask for a report of the final value of a test to the harness 
1. It serves to decide whether a test passed or failed. For 
example, a value of 0 will indicate a test passed, any other 
value indicating an error (e.g. failed, not run, or unresol- 
ved). Accordingly, in case of multiple test processes on 
35 multiple agents, each partial result can be combined by a 
syntax file and then a test result can be decided upon. It 
accepts a single return-only argument: 

* STATUS_CODE which designates any integer value, one of 
those signifying a test passed. 
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The master instructions may include other self explanatory 
instructions , like DELFILE, or CHECKFILE, and so on. The 
syntax files for the master language and the system local 
files are preferably stored in hard disk 14 of harness 1. 
5 Thus, it is possible to transfer files , unzip files , delete 
files, set specific environment parameters (or variables) , 
aind more generally do anything needed for running a test, in 
a target 2-i, under control of the harness 1. Moreover, with 
these master instructions , once a test sequence has been run, 
10 a result is decided on, and may be sent back to the harness 
1 for processing and, if desired, for storage in hard disk 14 
(Fig. 3). 

I • . 

The master instructions are used by the test parser for 
15 building up dynamically the desired environment (or test 
instruction execution configuration) adapted to data communi- 
cation between the harness 1 and each designated target local 
agent. This dynamic test creation is made and managed by test 
control module 6. 

20 

Thus, when writing the test, one does not need to know a 
priori what type of machine the test will run on. 

Moreover, the file transfer facility (SENDFILE) allows the 
25 release version which is to be tested, to be sent to the 

local agent machine 2-i as well, before executing the test. 

Thus, one no longer needs to manually re-install the local 

agent machine 6 with each new release version- Furthermore, 

since the local agent-hosting target communication is 
30 connectionless, even rebooting the target computer is 

allowed , as long as the local agent automatically restarts 

during the boot-up process. 

A non limiting example of a set of master instructions is 
35 shown as Example I in Exhibit A. It will send a file 
"passtest.zip" to a local agent , unzip it, run the Java class 
contained in this zip file "PassTest", and report the result 
to the test harness. Lines beginning with n # n are not 
executable. 
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It is assumed that a local agent " AGENT 1 " has been defined 
from an agent definition file 81. The agent definition 
includes a variable AGENT1 JDSNAME, containing the name of the 
OS in the corresponding target. The value of AGENT l_OSNAME 
5 may be "Windows NT" or "Solaris". 

Section A.l defines default hard-coded settings: the path to 
the Java home directory in the target; the path and name of 
the Java Interpreter in the target; a keyword representing 

10 the target OS class; the path to a temporary directory in the 
target; and a name TEST_NAME given to the current test 
sequence. In section A.l, the variable "FILE^SEPARATOR" is 
referred to the agent machine, seen from the harness. Java 
supplies another variable noted "file. separator" , for the 

15 local machine (whether harness or target). The default 
settings might be imported from one of the files 82, for 
example by using the instruction INCLUDE ( OTHER_FILE ) in lieu 
of section A.l. 

20 Section A. 2. sets up an agent to run the test on. 
"AGENT1_NAME" and "AGENTl_PORT" are variables previously 
defined by the user, via the harness interface. "AgentlD" is 
returned after execution of the instruction. 

25 Section A. 3 fetches a zip file "passtest.zip" containing the 
test, in a subdirectory of the harness, and sends it to the 
local agent in the target. "Passtest.zip" is one of the files 
labelled 62 in Fig. 6. 

30 Section A. 4 executes the test, using a timeout of 30 seconds. 
More precisely, it launches the Java virtual machine to 
execute the Java class "PassTest" unzipped from 
"passtest.zip" in the target temporary directory named 
"TMP_DIR". 

35 

Section A. 5 gets the status returned by the execute command. 
Section A. 6 gets the standard output and standard error 
traces. Section A. 7 cleans the standard output and standard 
error traces. Section A. 8 cleans the zip file on the agent 
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side. Section A. 9 clears any leftover agent files. 



Finally, section A. 10 reports the status to the harness user 
interface, e.g. a GUI. 

5 

The generic test code 850 of Fig. 5 is comprised of sets of 
instructions which are target independent, like the sets of 
instructions in Examples III and IV in Exhibit C. These 
examples are given only to illustrate what target independent 
10 code may be, and are not intended to const itue a part of an 
actual test code. 

The above discussed example shows how the sequence of test 
operations contained in the file "passtest . zip" will be 
15 executed in a target. As noted, the hard-coded setting 
variables for each target may be contained in a target 
dedicated file, which is simply called before execution of 
the test sequence itself. 

20 It should be now understood that the test parser is respon- 
sible for parsing the script 60, and for adapting the same 
according to the designated agent name and agent port number 
(the agent doublet of parameters). In the example of Exhibit 
A, after an agent is set, that agent is referred to by its 

25 AGENT_ID only. 

The general structure of a test specification (script) may 
include the following steps: 

- first, each local agent is queried for its operating system 
30 (OS) and hardware architecture. 

- then, some of the local agents are M SET" for execution of 
the test. The SETAGENT instruction is valid until a 
corresponding DELAGENT is executed. Note a DELAGENT simply 
means that the local agent is no longer seen from the harness 

35 resident test program. 

- then, using the AGENT_ID as necessary, the harness builds 
up the test commands which will be sent to each of the local 
agents involved in the test specification, for the test to be 
run. Each sent command has its own " Command I D n . 
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- at the end, the results are stored in 89 (Fig. 5). 

Accordingly, the script of master instructions may generally 
comprise any of the following, in any desired order 
5 (compatible with parsing requirements): 

- set initial condition statements 

- set variables 

- set properties 

- send commands to local agents 
10 - retrieve command results, 

decide whether the unitary test (set of master 
instructions) passed or failed. 
I 

The script is conveniently handled using a parser and editor 
15 based on a Graphic User Interface or GUI. It is also possible 
to run a test sequence without any GUI interface, if one 
makes use of a parser command-line interface, for example 
while specifying : 

* a session name which indicates the number of the run, which 
20 will be used to store the results retrieved from the local 

agent (s) in a dedicated home directory; 

* a property file which contains all the variables (or 
parameters) required to parse the test operations to be run; 

* a test path which specifies the full path to each syntax 
25 file to be run. 

Reverting to Fig. 5, hard disk 14 stores a collection of test 
specifications 841, corresponding to different levels, 
including: 
30 - OS level, e.g. Solaris or NT, 

- level of languages supporting the software to be tested, 
like Java 1.1 or Java 1.2 in the Java implementation, 

- level of the hardware architectures, like Intel X86, Alpha, 
or Sparc. 



35 



These various levels may be analyzed in a single test script 
by constructing a tree like optional structure, using e.g. IF 
statements, and sequentially considering the pertinent 
options in the above indicated levels. This is illustrated by 
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Example II in Exhibit B. Actions may be taken within the 
tree, or when reaching a leaf of the tree. Actions may 
comprise a CMD START using an adequate one of the command 
files 841. 

5 

The function of the test script (and of its master language) 
is to elucidate, based on the initial information known about 
each target agent (host and port number), enough information 
to send to the agent all information and/or data required to 

10 properly exercise the software being tested. This information 
is sequentially built up by dynamically querying the agent at 
the time of executing the test, and/or building on previously 
queried information and/or supplied default values (default 
properties) for each of the tested architecture/OS 

15 combinations. 

When parsing such a script, the test parser considers the 
local agent being set therein, and progressively builds up a 
test configuration adapted to that local agent. If several 
20 local agents are set, they are processed selectively the same 
way, with each agent finally receiving test commands adapted 
to its own architecture. 

After the parsing of all or part of the script, the logical 
25 harness 131 forms, in RAM 14, a set of agent specific 
commands, which are ready for being directly sent as requests 
to the local agent of a target 2-i, via sockets 17-i and 25- 
i. 

30 The above mentioned collection of test specifications 841 may 
be managed as a database, where each test specification is 
associated to its corresponding OS, supporting language, and 
hardware requirements. It may also include an identification 
of the corresponding specific software release version to be 

35 tested. 

Thus, in the script, most or all of the test operations 
themselves may be written in an agent-independent manner, 
with a syntax matching the harness test parser. 
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The "SETAGENT" portion of the script, and the corresponding 
analysis of "levels" may be considered as establishing a 
correspondence between active local agents and records in the 
above mentioned database. This correspondence may be 
5 materialized by a correspondence table between local agents 
(whose identifiers are defined) and corresponding test 
specifications (sequences of test operations) to be run with 
a release version in targets 2-i. 

10 For building up the test configurations, the harness 1 may 
also comprise a configuration file for each local agent 
registered in the correspondence table (or, at least, a 
configuration file for each local agent involved in the 
currently processed test sequences). 

5 

Generally, the test sequences in the harness include master 
instructions, the execution mode of which is stored in mass 
memory 4. The master instructions may have one or more 
arguments (or parameters). In fact, most of the master 
0 instructions are request instructions, i.e. will result into 
a request being sent by the harness to one of the local 
agents; again, in most cases, one argument (or more) in the 
request instruction is a command to be executed in the 
target, or includes such a command. 

5 

In case of Java language, all that is required to run any 
test sequence on any target computer is a Java virtual 
machine running a local agent in that target computer. 
However, the invention may be correspondingly extended for 
> implementation using other languages. 

Thus, to execute a sequence of test operations, residing on 
the harness 1, one just needs to configure the target 
specific elements of each supported agent machine architectu- 
re. The configuration values typically do not change after 
the initial set-up of the system. 

Moreover, the harness .1 may allow the user to effect 
permutations of the local agents, and since the local agents 
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can be queried, it is possible to detect, just prior to 
running, what the destination set-up (Java virtual machine, 
O.S., hardware architecture) needs to be, and to send the 
corresponding commands (and requests) comprising the test 
5 sequence to the local agent(s). 

It is now assumed that a current test combination is running, 
testing for example networking operations in a NT-Solaris 
environment (a target uses NT, another one Solaris). 
10 Switching to a Solaris-Solaris environment involves 
designating the local agents one wants to run the networking 
test on; everything else may be automatic. 

Men skilled in the art will understand that the invention 
15 applies independently of the particular "master language" 
used to write the instructions of the test sequence. The 
harness 1 may allow users to define their own shell script 
parser for use with the other testing tools. 

20 The parser in the given implementation of the technology is 
implemented as a Java class interface to allow abstraction of 
the actual parser details from the GUI, and to facilitate the 
implementation of other types of parser by simply 
implementing the provided interface "Parserlnterf ace" , shown 

25 in Exhibit D. Men skilled in the art will be able to 
implement their own master language syntax using this 
standard Java abstraction mechanism. 

If the technology was implemented in a language other than 
30 Java, a similar abstraction method would be used, such as a 
Corba IDL interface, or include file header. 

Whether the harness may run on Windows NT or Solaris or any 
other hardware /software configuration (or several of them) 
35 depends on how port ably the test sequences are written, as 
far as the master computer architecture is concerned 
("portably" refers here e.g. to the master instructions 
references to the auxiliary data files 62, and other platform 
specific properties required to test the software, such as 
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absolute file paths, drive letters, interrupts, etc). 

The approach of this invention greatly facilitates the 
distribution of tests, by centralising them in a single 
5 master computer (harness) and dynamically selcting and 
distributing these tests, when needed, to simple target local 
agents, which act as "mini-servers" waiting for commands to 
be run on the "target" primary computer provided with the 
software to be tested. 

10 

Thus, a kind of test database, associated with a number of 
different target computer architectures, is stored in the 
harness computer. Accordingly, a test operator will simply 
have to designate, in the database, one (or more) local 
15 agent (s) intended to run a selected test; after that, a 
specific test environment is dynamically created between the 
harness computer(s) and the designated local agent(s) in 
their respective hosting target computers. 

20 In other words, effectively running a software test sequence 
now consists in affecting "values" to the parameters (or 
variables) identifying every target local agent involved in 
that particular software test sequence. 

25 This database may be modified or completed with new test 
sequences every time a new computer architecture appears. 

The test system may be built as follows: 

30 - there is provided for primary computers 2-i having 
respective specific architecture (in terms of software and 
hardware). Each primary computer 2-i is provided with a soft- 
ware (or application) to be tested and at least one local 
agent. This software and the local agent(s) are adapted to 

35 their host primary computer architecture, and each local 
agent is capable of communicating data and running the 
software, on command. 

- provision is also made for a master computer or harness 1, 
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which may store a correspondence between local agents and se- 
quences of test operations to be run with a software to be 
tested in a primary computer 2-i. The or each master computer 
1 may also communicate with the primary computer local 
5 agent (s). 

In operation (Figure 8), the test personnel will: 

- designate at least one primary computer local agent, e.g. 
in a table (step 101). This may be implemented with a GUI. 

10 The designation consists preferably of a couplet of 
parameters (or variables) such as a name identifying the 
local ^agent primary computer and a port number identifying 
the TCP socket 5 attached to the local agent. 

- select each sequence associated with each designated 
15 primary computer local agent (step 103); 

- at step 105, the master computer builds up dynamically a 
specific test environment (or configuration) adapted to data 
communication between the master computer (s) 1 and each 
designated primary computer local agent. This may be done 

20 with a parser. 

- as a result (step 107), data representative of the selected 
sequence of test operations are communicated from the master 
computer 1 to the selected primary computer local agent (s), 
in the specific test environment, for running the desired 

25 test operations with the software to be tested, in the 
primary computer (s) 2-i. 

- at step 109, the harness gathers the results, i.e. reports 
sent back from the primary computer. 

30 The invention is not limited to the embodiments and processes 
described above, which are given solely by way of example; 
many alternative embodiments may be envisaged by men skilled 
in the art . 

35 For example, multiple local agents may reside in the same 
target, or may be spread across several targets, as long as 
all the doublets designating the local agents are unique. 
This may be of use for example when the release version to be 
tested is an application requiring results or information 
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provided by several targets. 

On another hand, the invention is not limited to local agents 
and/or harnesses written for implementation in a Java virtual 
5 machine . 

This invention also covers the proposed software code itself, 
especially when made available on any appropriate computer- 
readable medium. The expression "computer-readable medium" 

10 includes a storage medium such as magnetic or optic, as well 
as a transmission medium such as a digital or analog signal. 
The software code basically includes the code for use in the 
harness, and the code for use in the targets. It may also 
include the accompanying files, and further include at least 

15 some of the group of software components to be installed. 

The software code of a test program in the master test 
computer comprises: 

- code adapted for selective communication with a remote 
20 agent, 

code forming a test sequence module comprising a 
representation of a sequence of test operations, and 

- code forming a test control module adapted for sending 
commands derived from at least certain of the test 

25 operations, from said master computer to a selected local 
agent. 

The software code of a local agent comprises code capable of 
having commands to be run in an hosting computer, and capable 
30 of communication for receiving such commands. 
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EXHIBIT A - Example I 
#A.l 

JAVAHOME_NT= "c:\jdkl .2.2" 
5 JAVAHOME_SOLARlS="/usr/javal.2" 

IF (AGENT1JDSNAME == "Windows NT") 
THEN 

FILE_SEPARATOR = "\" 
10 JAVAJ^^^ERPRETOR=JAVAHOME - .NT+FILE_SEPARATOR+ w bin ,, + 

FILE_SEPARATOR+ "java.exe" 
KEYWORDS = "windows" 
TMPJDIR = "c:\tmp" 
ELSE 

15 FILE_SEPARATOR = "/" 

JAVAJNTERPRETOR= JAVAHOME_SOLARIS + FILJESEPARATOR+ "bin" + 
FILE_SEPARATOR+ "java" 
KEYWORDS = "solans" 
TMP_DIR « "Amp" 
20 END 

TEST_NAME = "Test pass example"; 

#A.2. 

SETAGENT(AgentID, AGENT1_NAME, AGENTl_PORT, TMPJDIR); 

25 

#A.3 

ZIPSEND(AgentID, zipID, 

TESTPATH + fi)e.separator+ "tests" + file, separator + "zips" +file.separator+ "passtest.zip"); 
30 #A.4 

CMDSTRING = JAVAJNTERPRETOR+" -classpath "+TMPJDIR+" PassTest"; 
CMDSTART(AgentID, CommandID, CMDSTRING, "30"); 

# A.5 

35 CM DSTATUS(CommandID, StatusCode); 
#A.6 

CMDGETTRACE(CommandID); 
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# A.7 

CMDCLEAN(CommandID); 

# A.8 

5 ZIPCLEAN(zipID); 

# A.9 

DELAGENT(AgentID); 

10 #A.10 

REPORTSTATUS(StatusCode); 



I 
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# OS level 

IF (AGENTl_OSNAME == "Windows NT") 
5 ... set corresponding variables 
MY_LEVEL = "NT" 
END 

.... (other cases of OS) 

10 # Supporting language level 

IF (TESTSUPPORT = = "Java 1.1") 

... set corresponding variables 

MY_LEVEL = MY_LEVEL + " Jl 1 w 
END 

15 .... (other cases of language) 

# hardware level 

IF (AGENT1_HARDNAME == "Intel X86") 
... set corresponding variables 
2 0 MY_LEVEL = MY_LEVEL + "X86" 
END 

.... (other cases of hardware level) 

# leaf level (final selection of test) 
25 IF (MY_LEVEL = = "NTJl 1X86") 

# do the same as A.4 in Example 1. This includes the CMDSTART in A.4, 

# applied to a command corresponding to that value of MY_LEVEL. 

# if necessary, A.4 is preceded by A.3, to send the file corresponding 

# to that value of MY_LEVEL to the target. 

30 # steps A.5 through A. 10 may follow, as necessary. 
END 

.... (other values of M Y_LEVEL) 
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EXHIBIT C 
Example III 

5 # Add up all odd numbers between 0 and 10 
ODDNUMBER.SUM = 0 
FOR I = 1 TO 10 BY 2 DO 

ODD_NUMBER_SUM += I 
END 

10 PRINT "Sum of odd numbers between 1 and 10 is " + ODD_NUMBER_SUM 

Which will generate the output : 

Sum of odd numbers between 1 and 10 is 11 

15 Example IV: 

# Generate a random number 
RANDOM(RANDOM_NUMBER) 

PRINT "We generated the random number :" + RANDOM_N UMBER 

20 

Which may generate the output : 

We generated the random number :54324554 

Other more sophisticated examples of such code may be generation of Fibonacci sequences, prime number 
25 generation. 
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EXHIBIT D - Methods of the PARSERINTERFACE class 
getSyntaxKeyWords 

public java.lang.Stringf] getSyntaxKeyWordsO 
5 This method lists all the keywords supported by this syntax, for use in the Notepad syntax 

highlighting. It must be declared static. 

getSyntaxSeparators 

public java.lang.String[] getSyntaxSeparatorsO 
10 This method lists all the separators supported by this syntax, for use in the Notepad syntax 

highlighting. It must be declared static. 

I • 
setTestPath 

public void setTestPathQava.Iang. String testFile) 
15 This method sets the path to file file containing the syntax which will be parsed 

setPrintStream 

public void setPrintStreamCjava.io.PrintStream printStream) 

This method sets the PrintStream to use for reporting errors and other types of output from the 
20 script. 

setProperties 

public void setProperties (Java. util. Properties p) 

This method sets any default properties which will be required for parsing this file. 

25 

getProperties 

public java. util. Properties getPropertiesO 

This method returns all the properties obtained by parsing this test file. 

3 0 setEvaluationMode 

public void setEvaluationMode(boolean evalMode) 

If set to true, the parser does not actually make contact with the agents but merely simulates the 
agent responses to allow standalone parsing. 

35 parseFile 

public int parseFUeO 

throws java. lang. Exception 
This method parses the specified file. If not in evaluation mode, it should return the status of the 
test run : Protocol Constants. PASSED Protocol Constants. FAILED Protocol Constants. NOTR UN 
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ProtocolConstants. UNRESOLVED 
interrupt 

public void intemiptO 
5 This method is responsible for killing any processes already started on the agents, and 

immediately halt parsing any files. 
getProperty 

public java.lang. String getProperty (java.lang. String key) 

This method retrieves the specified property from the results of parsing this file. 

10 

getProperty 

public java.lang.String getProperty (java.lang. String key, 

java.lang. String defaultValue) 

This method retrieves the specified property from the results of parsing this file. If the value is 
1 5 not found, the defaultValue is returned. 

getTestName 

public java.lang. String getTestNameO 

This method should return a test name which will be used to display the test in the test tree. 

20 

getKey Words 

public java.lang. String[] getKey WordsO 

This method should return all keywords associated with this test. These will be used in using the 
keywords to select/deselect tests in the harness. 

25 

getln elude List 

public java.lang.StringQ getlncludeListO 

This should return the list of files other than standard java.util. Properties files which were 
included to parse this test file. 

30 

getPropertiesIncludeList 

public java.lang. String[] getPropertiesIncludeListO 

This should return the list of standard java.util. Properties files which were included to parse this 
test file. 

35 

getTraceList 

public java.lang. String[] getTraceListO 

This method should list all available output files produced by this test when run on the agent, but 
relative to the harness. 
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1. A method of testing computer software, comprising the 
steps of : 

5 a) providing a primary computer (2-i) with a software to be 
tested/ 

b) providing the primary computer with a local agent (231) , 
the local agent being capable of having commands to be run in 
the local agent's primary computer, and having a 

10 communication interface, 

c) providing a master computer (1), arranged for selective 
communication with the communication interface of the local 
agent (231), with a representation of test operations being 
stored in the master computer ( 1 ) , and 

15 d) dynamically and selectively sending commands derived from 
at least certain of the test operations, from said master 
computer (1) to the local agent (231). 

2. The method of claim 1, wherein, in said master computer, 
20 the representation of test operations comprises data stored 

in correspondence with a designation of the local agent, and 
a sequence of instructions, at least some of which have the 
designation of the local agent as a parameter, 
and step d) comprises: 
25 dl) scanning the sequence of instructions, 

d2) for at least some of the instructions, building a test 
command from that instruction and the data corresponding to 
the local agent parameter of that instruction, and 
d3) sending the test command to the local agent. 

30 

3. The method of claim 2, wherein the designation of a local 
agent comprises a name given to the local agent. 

4. The method of claim 3, wherein the designation of a local 
35 agent comprises a port number assigned to the local agent. 

5. The method of claim 1, wherein step a) further comprises 
providing the primary computer (2-i) with a common software 
environment, and the local agent of step b) is a program 
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written for that common environment. 
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6. The method of claim 5, wherein the common environment 
comprises at least one Java virtual machine (230,130). 

5 

7. The method of claim 2, wherein the instructions are 
selected from a predefined set of instructions forming a test 
programming language , the predefined set of instructions 
comprising an instruction requesting a local agent to start 

10 execution of a command. 

8. The method of claim 7, wherein the set of instructions 
further comprises an instruction requesting a local agent to 
report on the status of execution of a command. 

15 

9. The method of claim 7 , wherein the set of instructions 
further comprises instructions for sending files to and 
retrieving files from a local agent. 

20 10. The method of claim 7, wherein the set of instructions 
further comprises an instruction for instituting a 
correspondence between a local agent name, a corresponding 
port in the master computer, and an agent identifier. 

25 11. The method of claim 7, wherein the set of instructions 
further comprises an instruction for storing data in 
correspondence with a local agent. 

12. The method of claim 7, wherein the set of instructions 
30 further comprises an instruction for sending at least a 

portion of the software to be tested to a local agent, for 
storage in the primary computer. 

13. A system for testing software in computer, said system 
35 comprising : 

- a primary computer (2-i), provided with a software to be 
tested, and with a local agent, the local agent being capable 
of having commands to be run in the local agent ' s primary 
computer, and the local agent having a communication 
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interface , 

- a master computer (1), having an interface (17) adapted for 
selective communication with the respective local agent 
interfaces, and having a memory comprising: 
5 . a test sequence module (8) comprising a representation of 
a sequence of test operations, and 

.a test control module (64,66,70) adapted for sending 
commands derived from at least certain of the test 
operations, from said master computer (1) to a selected local 
10 agent. 

14. The system of claim 13, wherein the test sequence module 
(8) comprises: 

. data stored in association with a corresponding local 
15 agent, and 

. a sequence of instructions, at least some of which have a 
designation of the local agent as a parameter, 
and the test control module comprises: 

. code (64) for scanning the sequence of instructions, 
20 . for at least some of the instructions, code (66) for 
building a test command from the instruction and the data 
corresponding to the local agent parameter of that 
instruction, and 

. code (70) for sending the test command to the local agent. 

25 

15. The system of claim 14, wherein the designation of a 
local agent comprises a name given to the local agent. 

16. The system of claim 14, wherein the designation of a 
30 local agent comprises a port number assigned to the local 

agent . 

17. The system of claim 13, wherein the primary computer (2- 
i) is provided with a common software environment, and the 

35 local agent is a program written for that common environment. 



18. The system of claim 17, wherein the common environment 
comprises at least one Java virtual machine (230,130). 



WO 01/71502 PCT/IBOO/00338 

34 

19. The system of claim 14, wherein the instructions are 
selected from a predefined set of instructions forming a test 
programming language , the predefined set of instructions 
comprising an instruction requesting a local agent to start 

5 execution of a command, 

20. The system of claim 19 , wherein the set of instructions 
further comprises an instruction requesting a local agent to 
report on the status of execution of a command. 

10 

21. The system of claim 19, wherein the set of instructions 
further comprises instructions for sending files to and 
retrieving files from a local agent. 

15 22. The system of claim 19 , wherein the set of instructions 
further comprises an instruction for instituting a 
correspondence between a local agent name, a corresponding 
port in the master computer, and an agent identifier. 

20 23. The system of claim 19, wherein the set of instructions 
further comprises an instruction for storing data in 
correspondence with a local agent. 

24. The system of claim 13, wherein the communication 
25 interface of each local agent comprises at least one TCP 

socket (25) . 

25. The software code of a test program in a master test 
computer , comprising : 

30 - code adapted for dynamic and selective communication with 
a remote local agent, 

- code forming a test sequence module (8) comprising a 
representation of a sequence of test operations, and 

- code forming a test control module (64-66) adapted for 
35 sending commands dynamically derived from at least certain of 

the test operations, from said master computer (1) to a 
selected local agent . 

26. The software code of a local agent in a computer, 
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comprising code (231) capable of having commands to be run in 
an hosting computer, and capable of communication with 
another process for receiving such commands. 
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